Multimodal Similarity and Categorization 1 Running head: MULTIMODAL SIMILARITY AND CATEGORIZATION Multimodal Similarity and Categorization of Novel, Three-Dimensional Objects

Similarity has been proposed as a fundamental principle underlying mental object representations and capable of supporting cognitive-level tasks such as categorization. However, much of the research has considered connections between similarity and categorization for tasks performed using a single perceptual modality. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or using more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In this study, we addressed these questions by having subjects explore novel, 3D objects which varied parametrically in shape and texture using vision alone, touch alone, or touch and vision together. Subjects then performed a pair-wise similarity rating task and a free sorting categorization task. Multidimensional scaling (MDS) analysis of similarity data revealed that a single underlying perceptual map whose dimensions corresponded to shape and texture could explain visual, haptic, and bimodal similarity ratings. However, the relative dimension weights varied according to modality: shape dominated texture when objects were seen, whereas shape and texture were roughly equally important in the haptic and bimodal conditions. Some evidence was found for a multimodal connection between similarity and categorization: the probability of category membership increased with similarity while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, dimension weights varied according to modality in the same way for both tasks. The study also demonstrates the usefulness of 3D printing technology and MDS techniques in the study of visuohaptic object processing. Multimodal Similarity and Categorization 3 Multimodal Similarity and Categorization of Novel, Three-Dimensional Objects The question of whether similarity can provide a theoretical basis for general categorization behaviour has been a source of heated debate in the field of cognitive psychology (Goldstone, 1994; Hahn & Ramscar, 2001). Critics of this idea have argued that the notion of similarity is vague and context-dependent, that it cannot explain category coherence, and that it does not account for the important role of theoretical knowledge in categorization decisions (Murphy & Medin, 1985). Nonetheless, similarity has served as the basis for a number of influential models of categorization (Rosch & Mervis, 1975; Medin & Schaffer, 1978; Nosofsky, 1992), which have been particularly successful in explaining classification of perceptual stimuli, including novel, 3D objects (Edelman, 1999). However, much of this work has been carried out within the context of perception involving a single modality, usually vision. Considering similarity and categorization within a multimodal context opens up a number of important questions: Are the similarities between objects the same when they are perceived using different modalities or by more than one modality at a time? Is similarity still able to explain categorization performance when objects are experienced multimodally? In a preliminary study (Cooke, Steinke, Wallraven, & Bülthoff, 2005), we showed how multidimensional scaling (MDS) techniques can be used to quantify differences in perceptual similarities when objects are perceived using touch and vision. In that study, subjects saw or touched novel, 3D objects which varied parametrically in shape and texture and then rated the similarity between object pairs. Using similarity as a psychological distance measure, MDS was used to visualize stimuli as points in multidimensional perceptual spaces, as for example in Shepard and Cermak (1973); Garbin (1988); Hollins, Faldowski, Rao, and Young (1993). We found that the relative importance of shape and texture in these perceptual spaces differed according to modality: Multimodal Similarity and Categorization 4 shape alone sufficed to represent the stimuli when perceived visually, while shape and texture were both required when the stimuli were perceived haptically. In the present study, we extend this line of research by adding a second task, free sorting categorization, and including a condition in which objects are simultaneously both seen and touched. The categorization task was included in order to test whether a connection between similarity and categorization could be established within this multimodal context. The bimodal condition was added in order to assess whether multimodal similarity and categorization would be dominated by one specific modality. At first glance, vision appears to be the most likely candidate. Vision is traditionally considered to be the ”dominant” modality (Rock & Victor, 1964). Furthermore, object shape has been shown to play a special role in category formation (Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976; Landau & Leyton, 1999) and shape is thought to be a particularly salient feature for vision (Klatzky, Lederman, & Reed, 1987). On the other hand, recent studies have challenged the notion of ubiquitous visual capture and have argued in favour of weighted averaging models (Guest & Spence, 2003; Ernst & Bülthoff, 2004). The results of this study show an effect of modality on the relative importance of object properties for both similarity and categorization tasks. In the bimodal condition, shape and texture were weighted roughly evenly for both tasks, rejecting the visual capture hypothesis. The probability of objects being grouped together in a category increased with similarity, while the probability of a category boundary being placed between two stimuli decreased with similarity. In addition, the relative importance of dimension weights for similarity and categorization tasks varied in the same way as a function of modality. The connection between similarity and categorization in the context of visuohaptic object processing is discussed in light of these findings. Multimodal Similarity and Categorization 5